Skip to content

Resolve glob matches in one path space (fixes #191) - #193

Merged
typeless merged 2 commits into
mainfrom
fix/191-glob-path-space
Jul 28, 2026
Merged

Resolve glob matches in one path space (fixes #191)#193
typeless merged 2 commits into
mainfrom
fix/191-glob-path-space

Conversation

@typeless

@typeless typeless commented Jul 27, 2026

Copy link
Copy Markdown
Owner

expand_glob_pattern merges two sources of matches: a filesystem scan, whose paths are source-relative, and a scan of Generated graph nodes, whose paths carry the build-root prefix. Out-of-tree those are different path spaces, and everything downstream compared them as plain strings — the sort, the dedup, and apply_exclusions.

Three symptoms, one cause

%f order depended on the name of the build directory. "../AAA/a/gen.dat" sorts before "keep.dat"; "../zz/a/gen.dat" sorts after:

putup -B AAA  ->  cat ../AAA/a/gen.dat keep.dat     all.txt: GEN\nKEEP
putup -B zz   ->  cat keep.dat ../zz/a/gen.dat      all.txt: KEEP\nGEN

A generated file shadowing a same-named source appeared twice — the halves spelled it a/x.dat and build/a/x.dat, so the dedup kept both and the checked-in source was never consumed:

cat ../build/a/x.dat ../build/a/x.dat > ../build/a/all.txt

Exclusions silently stopped applying. Normalized source-relative, compared against a build-prefixed path, so !skip.gen held in-tree and was ignored out-of-tree.

The fix

The loop already computed the build-root-stripped path to run the glob against, then discarded it in favour of the physical one. It now pushes the stripped path, so both halves are source-relative and the existing sort, dedup and literal exclusion comparison are correct as written.

resolve_input_node checks BuildRoot before SourceRoot at source-relative paths, so the stripped spelling still names the generated node — and the node is known to exist, since these matches come from iterating the Generated nodes themselves.

PR #180 introduced this merge and a follow-up normalized the ../ component lexically. That was one axis of a two-axis problem; this is the other.

Tests

Three e2e scenarios, each the shape that was never covered — a glob matching a generated and a source file, out-of-tree. Each observed failing first; the shadowing case showed FROMRULE twice.

Note the test-suite gap was narrower than I first wrote in the issue: out-of-tree coverage is extensive (265 scenarios use -B build). What was missing was specifically a glob spanning both node kinds, which is the only shape where the two spaces meet.

Also checked by hand, because the previous fix in this area got a related case half-right: a ../ pattern crossing directories out-of-tree resolves to distinct files in source-relative order, with correct physical paths rendered.

cat ../build/b/gen.q ../b/src.q > ../build/m/out.txt

Verification

make test: 142404 assertions, 637 cases, 32 e2e shards. make tidy, make iwyu clean. GCC BSP parses (24 Tupfiles, 1853 commands).

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA


Review corrections

An adversarial review found the claims here were broader than the fix, and one test asserted the wrong thing.

The exclusion axis is only half closed. A glob exclusion (!s*.gen) is expanded against the filesystem, so it never sees generated files and is still ignored out-of-tree; in-tree it is unstable, rendering echo keep.gen skip.gen on build 1 and echo keep.gen on build 2. Bin {group} members are still pushed build-prefixed into the same list. Both pre-existing, both verified on main too, now filed as #195. The comments in this PR no longer claim more than it does.

The order test had a blind spot. Asserting only that two build directories agree would still pass if the filesystem half stopped contributing entirely — both would be GEN and both equal. It now pins "GEN\nKEEP\n", covering source half, generated half and canonical order together.

The shadowing test canonicalized a divergence. Real tup rejects that project outright:

tup error: Attempting to insert 'x.dat' as a generated node when it already exists as a
different type (normal file).

and putup in-tree overwrites the checked-in source file on disk (FROMSRCFROMRULE). Filed as #194. The scenario is now tagged [deviation], states that it asserts only the dedup this PR is about, and points at the issue. Consequence worth noting: #191's second symptom is only reachable in a project #194 would reject, so that half of this fix is defensive rather than user-facing.

Behaviour change for existing projects. A Tupfile that worked around #191 by spelling an exclusion build-prefixed (!build/skip.gen) silently stops excluding. The source-relative spelling is the tup-conformant one, so the break is intended, but it is a break.

The review also verified independently, by execution, what I had argued: in-tree output is byte-identical to main (including a ../ cross-directory glob), the 3-tree GCC BSP dry run is byte-identical over 3530 commands, every consumer of these strings resolves BuildRoot-first before any exists() probe, an index written by main upgrades cleanly in one re-run, and all three new tests fail when the one-line change is reverted.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1076 M 0.51 s 14.2 k 0.8% 0.1% 0.493 s 31.1 MB (-0.1MB)
dry-run 1881 M 0.61 s 16.6 k 0.9% 0.1% 0.607 s 40.3 MB (+0.1MB)

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5818
Files changed 0
Files in index 6087
Graph edges 367913
Index size (bytes) 7378282
Implicit deps 330624
Hash computations 0
Hashes skipped (stat cache) 5818
Stat calls 5869
Parse time (ms) 478.1 (-1.5%)
Total time (ms) 585.7 (-0.9%)
Runner CPU AMD EPYC 7763 64-Core Processor

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.

Binary size (Linux)

Binary .text .data .bss File
putup 517.5 KB 1.6 KB 98.7 KB 615.2 KB

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
119 7503 6730 773 10.3%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 3571 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/cli/cmd_build.cpp 247
src/core/layout.cpp 69
src/core/new_delete.cpp 63
src/cli/context.cpp 55
src/graph/builder.cpp 44
src/core/string_pool.cpp 34
src/core/heap_buf.cpp 33
src/platform/vm-posix.cpp 22
src/cli/options.cpp 21
include/pup/core/layout.hpp 15

Test coverage (lines)

Overall Median file Min file Max file
86.7% 95% 0.0% src/platform/path-posix.cpp 100.0% include/pup/core/arena.hpp

103 files · 15164/17494 lines covered

Deltas vs main@618a6273f.

Updated for 485efc6

typeless and others added 2 commits July 28, 2026 11:59
expand_glob_pattern merges two sources of matches: a filesystem scan, whose paths
are source-relative, and a scan of Generated graph nodes, whose paths carry the
build-root prefix. Out-of-tree those are different path spaces, and everything
downstream compared them as plain strings -- the sort, the dedup, and
apply_exclusions. Three consequences, all reproduced:

  %f order depended on the name of the build directory, because
  "../AAA/a/gen.dat" sorts before "keep.dat" and "../zz/a/gen.dat" sorts after:

    putup -B AAA  ->  cat ../AAA/a/gen.dat keep.dat     all.txt: GEN\nKEEP
    putup -B zz   ->  cat keep.dat ../zz/a/gen.dat      all.txt: KEEP\nGEN

  A generated file shadowing a same-named source appeared twice, since the two
  halves spelled it "a/x.dat" and "build/a/x.dat" and the dedup kept both:

    cat ../build/a/x.dat ../build/a/x.dat > ../build/a/all.txt

  An exclusion, normalized source-relative, never matched a build-prefixed
  generated path, so !skip.gen was honoured in-tree and silently ignored
  out-of-tree.

The loop already computed the build-root-stripped path to run the glob against
and then discarded it in favour of the physical one. It now pushes the stripped
path, so both halves are source-relative and the existing sort, dedup and
exclusion comparison are correct as written. resolve_input_node looks under
BuildRoot before SourceRoot at source-relative paths, so the stripped spelling
still names the generated node -- and the node is known to exist, because these
matches come from iterating the Generated nodes themselves.

PR #180 introduced this merge, and a follow-up commit normalized the ../ component
lexically. That was one axis of a two-axis problem; this is the other.

Three e2e scenarios, each the shape that was never covered -- a glob matching a
generated and a source file, out-of-tree: order identical under two differently
named build directories, a shadowed file consumed once, an exclusion honoured.
Each observed failing first.

Also checked by hand, since the previous fix in this area got a related case
half-right: a ../ pattern crossing directories out-of-tree resolves to distinct
files in source-relative order with correct physical paths rendered.

GCC BSP parses (24 Tupfiles, 1853 commands).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
…test

From an adversarial review of this PR.

The comment claimed the exclusion axis was closed. It is closed only for literal
exclusions: a glob exclusion is expanded against the filesystem, so it never sees
generated files and is still ignored out-of-tree, and bin members are still pushed
build-prefixed into the same list. Both are pre-existing and now filed as #195.
The comments here now say only what this change does.

The order scenario asserted that two build directories produce identical bytes,
which would still hold if the filesystem half stopped contributing entirely --
both would be "GEN" and both would be equal. It now pins the exact content, which
covers the source half, the generated half and the canonical order together.

The shadowing scenario asserted putup's silent shadow-wins behaviour as correct.
Upstream tup rejects that project outright ("Attempting to insert 'x.dat' as a
generated node when it already exists as a different type"), and putup in-tree
overwrites the checked-in source file on disk. Filed as #194; the scenario is
tagged [deviation] and now states that it asserts only the dedup this PR is about,
not that shadowing should be legal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
@typeless
typeless force-pushed the fix/191-glob-path-space branch from e1f4f60 to 485efc6 Compare July 28, 2026 04:01
@typeless
typeless merged commit a1277e1 into main Jul 28, 2026
12 checks passed
@typeless
typeless deleted the fix/191-glob-path-space branch July 28, 2026 05:15
typeless added a commit that referenced this pull request Jul 28, 2026
A rule whose output path was also a checked-in source file was accepted, and in an
in-tree build putup overwrote the tracked file on disk:

  a/Tupfile:  : gen.src |> cp %f %o |> x.dat     (a/x.dat is checked in)

  before: FROMSRC
  after:  FROMRULE          <- the tracked file is gone

ensure_file_node silently upgraded an existing File node to Generated
(src/graph/dag.cpp:140-145), so the generated node simply won. Upstream tup rejects
the project instead, in-tree and in variant builds alike, and the diagnostic here
follows its wording including both remedies.

The bisect on the issue found no first-bad commit: this reproduces at the earliest
commit where a subdirectory rule can run at all, so there is nothing to revert and
the check has simply never existed.

What putup can decide, and what it cannot. The previous index is what tells a
source file apart from our own output sitting in the source tree after an in-tree
build -- the distinction tup's persistent node types carry. So the check fires on
exactly two answerable questions: with no previous index nothing we produced can be
on disk yet, so a file at an output's path is a source; with one, a path recorded
as a source File is a source whatever is on disk now. A generated file the previous
index does not mention is not decidable from here and is left alone -- that case is
real, not hypothetical: a scoped build drops out-of-scope commands from the saved
index, and the first draft of this check rejected the subsequent full build because
of it.

Deleting .pup while in-tree artifacts remain therefore errors until they are
cleaned. That is not a regression against tup: deleting .tup in the same situation
produces the same error and the same remedy, verified against the tup binary.

Config outputs are exempt. The two-stage configure design has a rule produce the
tup.config that the same build then reads as configuration, so that path is
generated and read on purpose rather than shadowing anything.

The [deviation]-tagged scenario added in #193 asserted the old shadow-wins
behaviour and said it should be rewritten if this issue were fixed by rejecting the
project. It now expects the rejection. The dedup #191 added for this shape is
consequently unreachable through a legal project and stays only as a defence, which
the test records.

Verified: rejection in-tree and out-of-tree with the source file intact; an in-tree
rebuild still succeeds with its own output on disk; scoped-then-full in-tree still
succeeds; GCC BSP parses (24 Tupfiles, 1853 commands) and busybox parses (31
Tupfiles, 293 commands).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant